Repository navigation
Reads treat values in dropped blob files as deleted - #15649
mikechuangmeta wants to merge 2 commits into
Conversation
Summary:
Changes
- Every compaction, automatic or manual, whose inputs may reference a blob
file recorded by `best_efforts_blob_db_repair` runs
`MissingBlobRepairIterator` in a new mode limited to recorded blob files.
Lost values become deletions at their sequence number; wide entities keep
surviving columns.
- These compactions run locally and skip garbage for recorded blob files.
Unrecorded missing blob files still fail with Corruption.
Why?
- After D131013277 records a dropped blob file, a compaction that drops one of
its values still fails: garbage for a blob file absent from the Version is
rejected ("unknown blob file"), and blob GC, filters and merges read the
missing payload. Reusing the `CompactRangeOptions` repair from D128836898
lets compactions proceed and removes the dangling references over time.
Changes visible to existing users
- None while no dropped blob files are recorded: each compaction adds an O(1)
check of its input Version's empty dropped set.
Design doc
- https://mdoc.internalmeta.com/doc/ffcd44f2-3147-4ced-aee4-c5623ccd6d8e
Reviewed By: xingbowang
Differential Revision: D131013279
Summary: Changes - Reads apply the compaction rule from D131013279 before fetching. In `GetContext::SaveValue` (Get, MultiGet, GetEntity, `GetEntityLazy`, `GetMergeOperands`, merges) and in `DBIter`, a value in a dropped blob file reads as a deletion at its sequence number; a wide entity reads as its surviving columns. - `MissingBlobRepairIterator` and reads share `IsDroppedBlobIndex` and `RemoveBlobColumns`. - New ticker `BLOB_DB_DROPPED_BLOB_READ_COUNT`. Unrecorded missing blob files still return Corruption. Why? - Reads returned Corruption for values in dropped blob files until a compaction rewrote the key, so the same key read differently before and after compaction. A missing blob file is lost data, not corruption. Reads now return what compaction will produce, including older versions under snapshots, merges over the lost value, and user-defined timestamps. Changes visible to existing users - None while no dropped blob files are recorded: reads add one branch per blob or entity entry, and `VersionBlobFetcher` construction checks the Version's empty dropped set. - New public ticker `BLOB_DB_DROPPED_BLOB_READ_COUNT`, C++ only because the Java ticker byte space is full. Known issues, for separate diffs on master - Tailing iterators fetch blob values through a Version they do not ref. This diff's own check uses a copy of the dropped list, so it is not affected. - `GetLatestSequenceForKey` merges SST merge operands into null outputs, which crashes for any base value. A dropped base now takes the same path as a deletion. Design doc - https://mdoc.internalmeta.com/doc/ffcd44f2-3147-4ced-aee4-c5623ccd6d8e Differential Revision: D131110165
|
@mikechuangmeta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131110165. |
✅ clang-tidy: No findings on changed linesCompleted in 669.5s. |
✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit f0b50b6 SummaryWell-structured change that makes reads consistent with compaction for dropped blob files. The design correctly handles Get, MultiGet, GetEntity, iterators, merges, wide-column entities, and timestamps. Test coverage is comprehensive. High-severity findings (0): Full review (click to expand)Findings🟡 MEDIUMM1.
|
| Context | Affected? | Assumptions hold? | Notes |
|---|---|---|---|
| WritePreparedTxnDB | YES | YES | Visibility via read_callback_ is independent of dropped blob checks |
| ReadOnly DB | YES | YES | Covered by test BestEffortsBlobDbRepairReadOnly |
| Tailing iterators | YES | YES | dropped_blob_files_ copied in constructor; test covers Version release |
| User-defined timestamps | YES | YES | Uses kTypeDeletionWithTimestamp; timestamp history test included |
| CompactionService | YES | YES | RepairsMissingBlobReferences() forces local compaction |
| Legacy BlobDB | N/A | YES | dropped_blob_files_ empty when expose_blob_index is true |
| GetLatestSequenceForKey | Partial | YES | SST lookup path doesn't pass blob_fetcher, so dropped blob entries are found as blob indexes (pre-existing behavior, noted in PR as known issue) |
| Concurrent writers | YES | YES | dropped_blob_files_ is immutable once Version is created |
Positive Observations
- The design reuses
IsDroppedBlobIndexacross reads and compaction, ensuring reads return what compaction will produce. RemoveBlobColumnsconsolidates duplicated entity column filtering into a single reusable function with a clean callback interface.- Copying
dropped_blob_files_for tailing iterators cleanly avoids Version lifetime issues. - Comprehensive test coverage: 10 new test cases covering Get, MultiGet, iterators (forward, reverse, tailing, unprepared), merges, wide-column entities, timestamps, auto-compaction, GC, read-only, and blob direct write.
UNLIKELYguards minimize feature-off cost on hot paths.RepairsMissingBlobReferences()efficiently checks only whether any input SST's oldest blob file could be in the dropped set, using the sorted property.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary:
Changes
GetContext::SaveValue(Get, MultiGet, GetEntity,GetEntityLazy,GetMergeOperands, merges) and inDBIter, a value in a dropped blob file reads as a deletion at its sequence number; a wide entity reads as its surviving columns.MissingBlobRepairIteratorand reads shareIsDroppedBlobIndexandRemoveBlobColumns.BLOB_DB_DROPPED_BLOB_READ_COUNT. Unrecorded missing blob files still return Corruption.Why?
Changes visible to existing users
VersionBlobFetcherconstruction checks the Version's empty dropped set.BLOB_DB_DROPPED_BLOB_READ_COUNT, C++ only because the Java ticker byte space is full.Known issues, for separate diffs on master
GetLatestSequenceForKeymerges SST merge operands into null outputs, which crashes for any base value. A dropped base now takes the same path as a deletion.Design doc
Differential Revision: D131110165